System and method for computer-assisted manual and automatic logging of time-based media

ABSTRACT

A customizable logging and content management system for indexing multimedia, including a synchronized timer object that provides a time reference upon request in connection with the media, and a logger object that logs predefined events that occur in the media by associating the events with respective time references from the timer object A video server is provided that captures and digitally stores events lugged by the logging application as media segments, and a search and retrieval engine is provided that enables that media segments to be located, retrieved and viewed based on the indexes. The system includes a graphical user interface generator that enables customized user interfaces and logging databases to be created from database tables for use in the logging application.

RELATED APPLICATION

[0001] This application claims the benefit of U.S. ProvisionalApplication Serial No. 60/168,216 filed Nov. 30, 1999, and entitled “AProcess for the Computer-Assisted Manual Logging of Time-Based Media”,the entire content of which is hereby incorporated by reference herein.

FIELD OF THE INVENTION

[0002] The instant invention relates to a multimedia asset managementsystem for use in managing audio, video and/or multimedia informationand assets. More particularly, the invention relates to a multimediaasset management system that enables capturing, indexing, cataloging,rearranging, safeguarding and redistribution of multimedia assets. Thesystem provides, among other things, a computer assisted logging systemfor use in logging and capturing audio and/or video segments from a liveor prerecorded event, such as a sporting events or any other time-basedevent that uses audio and/or video recording or broadcasting.

BACKGROUND AND SUMMARY OF THE INVENTION

[0003] Many industries such as the sports and entertainment industryrequire the ability to search through their archive of audio, videoand/or multimedia assets and find particular prerecorded information. Inthe sports industry, this search and retrieval ability is needed formany purposes. For example, when scouting for new players it can be veryuseful to have the ability to search through an archive of recordedgames to find particular events that occurred during the games. Forinstance, a coach may desire to review every three-point shot that abasketball player has attempted in a game or throughout multiple games.Also, a sports league may desire to collect and review many differenttypes of events that have occurred during the course of game play. Inthe entertainment industry, producers need access to archival footagefor highlights and many other reasons. For example, a sports televisionshow may desire to find and retrieve particular video segments from aparticular prerecorded event to show to the public during a newsbroadcast. For example, when broadcasting sports news, the network maydesire to show every video segment from a prerecorded football game thatshows a turnover of the ball from one team to another. In fact, insports, whether from the team, league, entertainment or other relatedperspective, there are numerous reasons why it is important to have theability to find and retrieve video and/or audio segments from an archiveof prerecorded information (e.g., video, audio and/or multimediaassets). While the ability to manage an archive of recorded assets isparticularly useful in connection with sports, such management systemsare needed for many other applications as well. In fact, the needextends to practically any situation where significant amounts of audioand/or video is recorded, such as in the movie production business,where one might desire to find and retrieve a particular segment ofinformation (audio, video or multimedia segment).

[0004] One technique that has been widely used in the past, particularlyin connection with sporting events, is to use manual loggers to loginformation about an event that is being recorded or has been previouslyrecorded. This technique involves using people, known as “loggers”, toview the entire event as it is being recorded (or after it has beenrecorded) and to index certain pre-defined events that occur during theevent that may be of interest to the person or organization for whichthe logger is working. For example, a logger working in connection witha basketball game may have the job of logging information, such as timeof occurrence during the game, that particular events occur, such asfouls, three-point shots, defensive plays, or any other event that maybe of interest in connection with the basketball game. The loggertypically uses a log sheet on which he manually enters the event and thetime of occurrence of the event for later reference. Numerous loggersmay be used for any one sporting event. When the loggers are finishedlogging the event, the log sheets are collected and stored with therecording of the event, so that they can be used as a reference to laterfind particular segments of interest from the overall recorded event.For example, the log sheet may indicate that a particular basketballplayer made a three-point shot at a particular point in time during thegame. Thus, if a person wanted to see the video segment or clip of thisparticular three-point shot, the log time indicated on the log sheet canbe used to find the point in the recorded game where the shot occurs.Thus, the log sheets act as an index to particular segments within theoverall recorded event that may be of interest at a later time. Oneproblem, however, with this manual logging process is that it is timeconsuming and inefficient, in that the physical log sheets must bereviewed manually in order to find the information or event of interest.Moreover, once the index is found on the log sheet, the tape or othermedium on which the event has been recorded must by obtained so that theparticular video and/or audio segment can be clipped therefrom. Thus, amore efficient system is needed in connection with the logging of timebased media, such as sporting, entertainment or other events.

[0005] Video editing software exists for enabling editing of recordedvideo or multimedia assets. However, such video editing software is verysophisticated and generally requires a knowledgeable and experiencedvideo engineer to properly edit or obtain video clips from a video. Inaddition, such video editing software is expensive to purchase andoperate. Thus, a less expensive and more user friendly video segmentindexing and capturing system that can be used in connection withlogging and retrieval of time-based media is needed.

[0006] The instant invention addresses the above-described needs byproviding a computer-based indexing and capturing system for use inconnection with logging of audio, video and/or multimedia events. Theinvention provides a user friendly tool for use by loggers or the likeworking in connection with a time-based media event, such as but notlimited to a sporting event. Specifically, the instant invention enablesloggers to easily and efficiently view a live or prerecorded event anddocument a time-based stamp for predefined events that may occur duringthe overall event. In accordance with the invention, a user-friendlygraphical user interface (GUI) is provided that is customized to theparticular application (and optionally for each particular logger usingthe system) in which the system is being used. The GUI includes userinterface objects, such as check boxes, radio buttons, push buttons andtext boxes, that are used by the logger to record events. The interfaceobjects are predefined and customized for the particular asset beinglogged. In other words, the GUI objects are defined so as to correspondwith the typical types of events that are generally of interest for theparticular overall event that is being logged. In addition, the GUIobjects are labeled in a manner that corresponds to the typicalnomenclature of these events, thereby facilitating easy event logging bya logger through the GUI. The logged events are then stored in adatabase for later search and retrieval as desired. Thus, the inventionenables parsing, cataloging, storing and retrieval of time based media.

[0007] In one embodiment of the invention, the system provides a linearvideo cataloging system designed to enable easy and efficient logging orindexing of linear (analog) video content. In accordance with thisembodiment, the system enables a digital directory of linear videocontent to be created which facilitates precision retrieval ofhighlights and clips from recorded videos.

[0008] In accordance with another embodiment of the invention, a mediamanagement system is provided that accepts analog or digital videocontent, indexes the video content, indexes each video frame andprovides for advanced retrieval of video segments of interest. Thus, inaccordance with this embodiment, when a logger logs an event the logginginformation (index) and the actual video segment are captured by thesystem, thereby enabling the system to be used to search for, retrieveand actually view a logged video segment. As a result, this embodimentprovides a complete video asset management system that can be used tolog events and search for and retrieve the actual video segments thathave been logged using a dedicated search engine.

[0009] In accordance with another embodiment of the invention, thesystem is: provided with an automatic logging capability, whereinaudio/video analysis technology is employed for speech, sound, object,and text recognition to automatically index video content and create anelectronic directory of same. In other words, in this embodiment, manualloggers are not required because the system is programmed toautomatically log certain predefined events using audio/video analysistechnology.

[0010] The invention may be implemented in a stand-alone fashion, or itmay be deployed as a client-server application for a LAN or web-basedenvironment. The system is designed to enable multiple loggers to beoperating on the same overall event at any one time. Facilities areprovided for enabling all of the logged events from all of the loggersto be loaded into a common database for later search and retrieval.

[0011] In accordance with another aspect of the invention, the system iseasily customizable for any event that needs logging. Specifically, theGUI can easily be customized to fit any particular application. Inaccordance with another aspect of the invention, a user interfacegenerator is provided to facilitate the creation of customized, HTML-orXML-based user interfaces for the collection of standardizedinformation.

[0012] In accordance with another aspect of the invention, a videocontent management system is provided wherein the end-user can performcomplex search functions, such as search by film name, topic, bookmarks(using keywords), user groups, film groups, etc., and in responsereceive the exact desired content (film, scene or frame). This featureis designed to transform viewing habits and the manner in which peopleapply the use of video for work, study and entertainment. A video serveris at the heart of the video content management system. The video serverstores, manages and provides access to data in the form of large andcomplex video content files.

[0013] As will be apparent from the description below, one primaryobject of the instant invention is to simplify and speed up the processof manually logging audio and/or video or other time based media from alive event or a previously recorded event. The speed at which a person(the logger) can do the logging is especially critical when logging alive event, where there is limited time to perform logging functions.Another primary object of the invention is to provide a complete videocontent management system that not only enables efficient logging, butalso provides a sophisticated search and retrieval function. It is notedthat audio, video and other time-based media may simply be referred toas media in the description below, and that the invention is not limitedto any particular media type.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] These and other objects, features and advantages of the instantinvention will become apparent from review of the following detaileddescription of the invention when viewed in conjunction with thefollowing drawings, in which:

[0015]FIG. 1 illustrates the timer object and logging object inaccordance with one embodiment of the instant invention;

[0016]FIG. 2 shows an exemplary and greatly simplified graphical userinterface for logging events in accordance with the instant invention;

[0017]FIG. 3 shows a basic flow diagram of the logging process inaccordance with one embodiment of the instant invention;

[0018]FIG. 4 shows an exemplary schematic diagram of an overall systemin accordance with the instant invention;

[0019]FIG. 5 shows an exemplary schematic diagram of the main componentsof the system of FIG. 4;

[0020]FIG. 6 is a block diagram of the video server architecture inaccordance with the instant invention;

[0021]FIG. 7a shows the video content management system architecture inaccordance with one embodiment of the instant invention;

[0022]FIG. 7b a signal diagram of the video content management system ofFIG. 7a;

[0023]FIG. 8a shows an exemplary flow chart of the manual loggingfeature of the instant invention;

[0024]FIG. 8b shows an exemplary flow chart of the automatic loggingfeature of the instant invention;

[0025]FIG. 8c is a schematic of the video indexing in accordance withthe instant invention;

[0026]FIG. 8d is a flow chart of the search and browse feature of theinstant invention;

[0027]FIG. 8e shows a flow diagram of the video server;

[0028]FIG. 8f shows a flow diagram of the video analysis subsystem usedin accordance with the preferred embodiment;

[0029]FIG. 8g shows a preferred flow diagram of the application server;

[0030]FIG. 9 shows an exemplary video frame indexing format inaccordance with one embodiment of the instant invention;

[0031]FIG. 10 shows an exemplary relationship between the video index,features and event database;

[0032]FIG. 11 shows a block diagram of the media pump in accordance withone embodiment of the instant invention;

[0033]FIG. 12 shows an exemplary flow chart of the logging andapplication web-based user interface, in accordance with one embodimentof the instant invention;

[0034]FIGS. 13 and 14 show exemplary database tables used in accordancewith one embodiment of the instant invention;

[0035]FIG. 15 shows the resulting output of the GUI generator using thetables of FIGS. 13 and 14;

[0036]FIG. 16 shows a high level architecture of the GUI generator andGUI applet in accordance with one embodiment of the instant invention;

[0037]FIG. 17 shows an overall flow chart of the operation of the GUIgenerator and GUI applet in accordance with one embodiment of theinstant invention; and

[0038] FIGS. 18-46 show exemplary screen shots for an exemplarybasketball implementation of the instant invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] Various features and embodiments of the invention will now bedescribed with reference to the appended drawings. It is noted thatthese embodiments are only exemplary and that the claimed invention isnot meant to be limited to the specific embodiments described below.

[0040] An event, such as a sporting event or other time-based event, maybe recorded in a variety of ways. For example, it may be recorded to oneor more tapes. Alternatively, it may be digitally captured, possiblycompressed and stored on some form of digital storage. The instantinvention is designed to provide a management system for all types ofrecorded information, regardless of the manner in which it is typicallyrecorded and stored.

[0041] In accordance with a first and simplified embodiment, theinvention includes a timer object and a logging object, as indicated inFIG. 1. The timer object keeps track of the current time code, or timestamp, of the particular media being logged. The time code can be basedon either a relative or an absolute time. This time code is later usedin accessing the media to jump to the correct location within an event.The logging object includes a variety of user interface objects (such ascheck-boxes, radio buttons, push buttons, text boxes and/or the like).The labels on the user interface objects provide access to and loginformation in a database of logging information, as indicated in FIG.2. The logger chooses the desired objects that are to be associated withthe current time by, for example, clicking on the objects. For example,as seen in FIG. 2, the logger may select the “player 1” and “2-point”buttons to log the fact that player 1 has made a 2-point shot in abasketball embodiment of the invention. The system would then record,using the timer object, the particular point in time (relative orabsolute) when this event occurred in the course of the overall event,thereby providing an index to the overall event for finding the logged2-point event at a later time, if desired. Thus, when an object isselected it acts as an index to the media for the event being logged. Inother words, the logger object is operable to create and store thenecessary information that would enable someone to locate the event thathas been logged within the media. Thus, the invention eliminates theneed to perform manual logging on paper.

[0042] An exemplary process that is followed in order to log an event isshown in FIG. 3. The first thing that happens is that both the timerobject and the logging object are loaded. The timer object then waits tobe started, either manually by the logger or automatically as a triggerfrom some-external event. If an absolute time-of-day timestamp isneeded, the object sets the computer's current time of day from anatomic clock or other defined clock. The system then proceeds through aloop. The loop checks to see if: 1) the logging process is over; 2) atape has been changed (in the case of recording to tape); and 3) if auser has added an event. If the logging process is over, the processends. Otherwise, if the tape has been changed, the time code isoptionally reset to 0 and the tape number is incremented. In anotherembodiment, the time code is not reset, because the time may run fromthe beginning of the first tape when multiple tapes are used to recordthe event. If a user adds an event, the timer object is queried to getthe time stamp and tape number. This information is stored in adatabase. As indicated by the formula below, the time code may be storedas the current value of the time code minus some small period of time(delta), since by the time the button is pressed by the logger the eventmay have already passed:

T _(stored) =T _(pressed) −Δ

[0043] wherein delta is a small period of time, perhaps 10 seconds,depending on the complexity of the information being logged.

[0044] The timer object maintains the current time code. This time codemay be expressed differently, depending on the manner in which the mediais being used. For example, the time code may be expressed in twoways—it may be expressed as the difference between the current time andsome arbitrary time relating to the event (such as the beginning of theevent), or it may be expressed as an absolute time in terms of thecurrent time of day, synchronized with, for example an atomic clock.

[0045] The invention can be used for live logging with digital mediacapture. When capturing a live event digitally, the timer object wouldalso be programmatically tied into the digital capture of the media. Itis this capture process that knows how long the capture has beenproceeding. In this case, the time stamp is simply the amount of mediathat has been captured, expressed in time. The trigger to start thecapture (and therefore the timer object) can be manual, requiring loggeraction, or could be automatically generated from an external trigger orsignal.

[0046] The invention can also be used for live logging without digitalmedia capture. When capturing a live event non-digitally, the timerobject would simply be a timer that keeps track of the amount of timethat the media has been recorded, being careful to exclude any amount oftime where recording was suspended. In the case of tape-based recording,the timer object must also maintain a count of the number of tapes beingused. When the timestamp is stored to the database, the time and thecurrent tape number are added to the database. The trigger to start thetimer and also the trigger to switch tapes can be manual, requiringlogger action, or could be automatically generated from an externaltrigger or signal.

[0047] The invention can also be used for post logging with digitalmedia capture. When logging a digitally captured event after it has beencaptured (e.g., prerecorded), the timer object is a software digitalmedia playback object, such as Windows Media Player. The video is playedback and the play back object is queried for the current time positionby the logging object to be stored in the database.

[0048] The invention may also be used for post logging without digitalmedia capture. When logging a non-digitally captured event after it hasbeen captured, the timer object is a timer that keeps track of theamount of time that the media has been playing back, being careful toexclude any amount of time where recording was suspended. In the case oftape-based recording, the timer object must also maintain a count of thenumber of tapes being used. When the time stamp is stored to thedatabase, the time and the current tape number are added to thedatabase. The trigger to start the timer and also the trigger to switchtapes can be manual, requiring logger action, or could be automaticallygenerated from an external trigger or signal.

[0049] With respect to the logging object, each user interface objectcorresponds to a data field that may be logged. These data fields areretrieved from a database of frequently logged fields. In accordancewith the invention, this database is completely customizable, as will beexplained in detail below. The user interface objects are groupedhierarchically to make finding them easy. For example, as indicated inFIG. 3, when logging a sporting event there could be a column of userinterface objects for the players on one team (“Team 1”), another columnfor the players on the other team (“Team 2”) and a third column for thetype of event to be logged (2-point, 3-point or foul, using basketballas an example).

[0050] Preferably, the user interface objects are customizable on thefly, as well. For example, a blank field at the bottom of each groupingmay be provided to allow the logger to type the customized field, addthe field to the database of fields, and log the event using thatcustomized field. The choice of user-interface object associated with adata field defines the relationship between the data. Check boxes may beused to allow for multiple selections within a group. The ability to logmultiple players for an event is an example of where a checkbox is used.On the other hand, a radio button allows the selection of only one itemin a group. The type of event fields are an example where only one itemwould be selected at a time. To facilitate the speed of the logger, whenappropriate, the most commonly logged item in a group will be selectedat the beginning of logging a new event. For instance, in basketball, a2-point shot may be the most common event.

[0051] In accordance with the invention, once the media has been logged,a search application (search engine) is provided for accessing thelogged video segments, either digitally or otherwise. The user may entersearch information (i.e. player and type of play) as search fields. Theapplication then queries the database of stored events and creates alist of events that match the criteria. If the media is digitized andaccessible, the database will also provide instant access to that media.If the media is not accessible, it provides the location of the videoclip, in the form of a tape number and time code. In this manner, thelogged media segments can be easily located and retrieved using theinstant invention.

[0052] The invention may be implemented in a Web based embodiment.

[0053] Although it is possible to use this invention in any softwareenvironment, a web-based implementation has some advantages. Forexample, it may be setup as a convenient subscriber-based system. In aweb-based implementation running on a Web Browser, the timer object mayreside in one frame, while the logging object is in a different framewithin the same window. When the logger clicks a submit button, thelogging object frame refreshes from the web server, while the timerobject frame remains un-refreshed, since refreshing it would restart thetime object and lose the timing information. Sample code for accessingthe time object from the logging object frame is as follows:

[0054] window.parent.frames (0).TimerObject.CurrentPosition

[0055] The refreshed logging page has all but the default check boxesand radio buttons cleared, as well as a recap of the previous item addedto the database.

[0056] The above description has provided a general overview of the mainfeatures of the logging aspect of the instant invention. Additionalexemplary embodiments and further features of the invention will now bedescribed in connection with the remaining drawings.

Preferred Implementation Details

[0057] Generally speaking, this invention provides a multimedia assetmanagement system for individuals and enterprises that manage largevolumes of video and multimedia information and utilize such informationas an integral part of their business operations, or whose core businesscan be enhanced through the effective management of multimedia content.In other words, the invention provides a business solution toinformation-intensive companies who require capturing, indexing,cataloging, rearranging, safeguarding, and redistributing of theirmultimedia assets. This system does not address digital image creation,manipulation, and editing, nor does it enable digital artists to createimagery or transform graphical art for visual entertainment. Instead,the system is designed to manage digital assets. Specifically, thesystem is designed to parse, index, store, and re-broadcast compressedvideo information over, for example, the Inter-net, virtual privatenetworks, and 100 base-t local area networks. The basic functions of thesystem include:

[0058] 1. Capturing linear (analog) and non-linear (digital) videocontent;

[0059] 2. Encoding the video content into MPEG format;

[0060] 3. Indexing each video frame, creating an electronic directory;

[0061] 4. Storing the video content as to facilitate customizedretrieval, and

[0062] 5. Re-broadcasting compressed video in digital quality.

[0063] The system preferably manipulates and manages video informationwhile in a compressed form. This minimizes the storage and processingrequirements of the application while improving the quality of the videocontent. The system is operable to dynamically construct and maintain anelectronic directory by accepting user-generated or system-generatedreferences. Object recognition, speech recognition, and text recognitionmay be incorporated to enable the system to automatically createdetailed directories of indexes to the content.

[0064] The invention uses a broad range of enabling technologies thatfacilitate the creation, management, and commercialization of a digitalvideo library (DVL). The system may be designed as a client-serviceapplication that can be deployed in a LAN or Internet environment.Alternatively, the system may be implemented in a stand-aloneembodiment.

[0065] The system is preferably highly modularized and is divided intosix major sub-units: Capture Bridge, Media Pump, Database Mechanism, WebServer, Client Management, and Operation Support System (OSS). The OSSintegrates the overall system by logically associating all services andapplication management into a single environment. The application ispreferably designed to accommodate different market segments.

[0066]FIGS. 4 and 5 provide two illustrations showing the overall systemin accordance with a preferred embodiment of the instant invention. Thesystem includes a capture bridge for capturing media, such as bysatellite, video camera, video recorder/player or other device. Thiscapture bridge is the hardware abstraction layer (HAL) forhardware-based encoders. It enables the utilization of a variety ofthird parties' hardware encoders in addition to enabling a singleencoder to be shared across the network through COM and CORBA. Thesystem supports simultaneous video streaming requests from multipleclients through a single decoder (using COM this would be done usingConnection Point Interfaces). As can be seen in FIG. 5, the capturebridge includes a capture bridge administration module. ThisAdministration utility is a web-based application that can run remotely.It allows an administrator to control critical encoding features.

[0067] As can be seen in FIGS. 4 and 5 another main component is themedia pump. A block diagram of the media pump is shown in FIG. 11. Thismodule preferably interfaces with most of the third party video serversalready in the market, by using the URL format to specify the path to afile resource. These include (Real Networks, MS NetShow Theater). Themedia pump includes an application server, a security module, a storagemodule, a video server, a logging application, and a custom reportingmodule. As indicated in FIG. 4, many of the elements of the systememploy 3^(rd) party software. However, the video server and loggingapplication have been developed specifically as part of the instantsystem. Thus, these two parts will be described in greater detail below.

[0068] The video server preferably has the following features:

[0069] Connection setup upon client request;

[0070] Validate multimedia request with security component;

[0071] Generate stream from multiple sources as specified in clientrequest;

[0072] Stream multimedia content specified by client;

[0073] Receive incoming stream and store on file system;

[0074] Cross platform.

[0075] The video server is preferably administered through a web-baseduser interface that can be used from a remoter host.

[0076] The video server is the heart of the video content managementsystem. A video server is in some ways similar to—and in many ways quitedifferent from—the familiar file server used for management and storageof programs and data files in a personal computer network environment.Like a file servers the video server must store, manage, and provideaccess to data. The video server is different, however, in that it muststore and manage inordinately large and complex video content filesrather than small, simple programs.

[0077] The video server system preferably satisfies the followingrequirements:

[0078] 1. Can deliver a different video stream or application for eachuser.

[0079] 2. Can allow all users to access a single video stream orapplication.

[0080] 3. Can deliver any requested item at any time to any user,without a significant delay.

[0081] 4. Can deliver over one hundred digital video streamssimultaneously.

[0082] 5. Provides broadcast-level quality with industry-standard-formatdigital video (this implies 6 Mb/s MPEG-2 streams).

[0083] 6. Stores over one hundred video streams, each of which hasseveral hours of contents.

[0084] 7. Delivers the contents, in digital format, through an Intranetor Internet network.

[0085] 8. Can be expanded, both with regard to throughput and contents,without significant changes.

[0086] 9. Provides a variety of types of multimedia applications.

[0087] Requirements 1-3 imply that the system must be able to deliverthe full contents of any video file at any time. Requirements 4 and 5mean that the system must have an aggregate bandwidth of over 600 Mb/s.Requirements 5 and 6 mean that the system has over half a terabyte ofstorage. Requirements 5 and 7 imply that the contents must be encoded in6 Mb/s MPEG-2 streams with an MP@M (Main Profile@Main Level) formatdefined in the standard for digital broadcasting and must be deliveredwithout any conversion, such as digital to analog, in the transmissionstage. Requirement 9 means that in addition to providing file transferfrom server to clients, the system must have application protocols forinteractive operations between clients and server.

[0088] The preferred video server architecture is shown in FIG. 6. Thecore techniques are the stream deliver and stream control. They involvetwo interface devices, RAID controller (Real-time file interface) andnetwork controller (network interface).

[0089] In general, there are two schemes for transmitting data streamsfrom server to client: pull and push. The former is based on packettransmission with appropriate flow control between client and server.For instance, the TCP/IP protocol is a pull scheme, which can transmitdata with no loss of packets. This protocol requires processing powerfor dealing with protocol-stack handling. Many difficulties exist. Forexample, if a CPU is switched to another task, the transmission processis interrupted, and the stream is not delivered with continuity. In theworst case, the user sees a broken video image on the screen of theclient, even though the packets of the stream are not lost. Thissituation will be even more serious for higher-bit-rate streams, sincedata of the stream in the decoder of the client are consumed at the bitrate specified during the encoding process, and the transmission of eachpacket of the stream must keep up with the decoder's rate ofconsumption. In addition, the server must support multiple userssimultaneously. Thus, the overhead of protocol processing is higher atthe server side, and it is difficult to maintain the data transfer atthe required rate.

[0090] On the other hand, the push scheme is similar to a broadcastingapproach. Once a stream file has been opened with a specified transferrate, the server keeps transmitting the stream data at that data rate.Since there is no flow control between server and clients in thisscheme, the processing power needed for the stream can be minimized atboth sides. This means that the server can support the transmission of ahigher-bit-rate stream for many users and can continuously supplystreams of data to the decoders of clients. In order to meet therequirements described in the previous section, the instant inventionpreferably uses the push scheme.

[0091] As for the data format of the stream, the MPEG-2 transport streamis preferably used in order to minimize the overhead of the transmissionwhile keeping the original quality, since the stream consists offixed-size packets and its transmission is based on synchronizationbetween the sender and the decoder of the recipient. Furthermore, thisformat allows multiple streams to be multiplexed naturally into a singlestream. Therefore, the format is suitable for the transmission ofdigital video in a broadcasting mode. On the other hand, transmitting anMPEG-2 “program stream,” which is suitable for use with storage medialike DVD (digital video disc), requires appropriate flow control, inorder to avoid underflow in the decoder buffer. This is more difficultto manage for large-scale video delivery.

[0092] The instant video content management system requires mechanismsfor controlling the stream in various ways, with VCR-like (videocassetterecorder) operations at any time and with no significant delays.Therefore, the control path between client and server should befull-duplex in order to maintain interactive operations. This pathenables communication between client and server through use of theTCP/IP protocol.

[0093] In a push scheme, a stream of video (MPEG-2 transport stream) isdelivered continuously from server to client. This unidirectionalcommunication path may be referred to as the “down channel”.

[0094] There are several ways for handling requests from clients. Themost frequently used are the queuing method and the RPC (remoteprocedure call) method. Though the former method is rather simple andmakes it easy to isolate the client and the server, a bottleneck ofrequests can occur when the number of clients is large. If some user'srequest takes a long time for processing, the request queue for clientrequests can fill up, and the server may not be able to respond to theclients quickly. On the other hand, the RPC model establishes a pathbetween the server process and each client process, and control iseasily passed from the client process to the server process; thus,users' interactive operations can be dealt with without significantdelay. As a result, the instant system preferably uses the RPC model.

[0095] As for controlling streams and application objects, the systempreferably uses the command set of DSM-CC, a part of the MPEG-2standard. This command set supports not only stream-control operationsand file-related controls, but also application-protocol managementbetween client and server. It covers the requirements of a variety ofmultimedia applications in this interactive television system.

[0096] The video content searching algorithm is divided two parts:searching based on manual logger and searching based on automaticlogger. The first algorithm is based on searching in an index database,and the second is based on the voice and image recognition technology.In the first searching algorithm, the event searches in the indexdatabase, but the time-code comes from I frame detection. That means allclips start from an I frame and stop on an I frame.

[0097] The index database shows the relationship between an event and atime-code of the video stream. In the automatic logger there is afeature database which shows the relationship between an event andfeatures (such as: caption, shape, color, motion and so on). Thesearching event is converted searching features and gets the relativetime-code. FIG. 8a shows an example flowchart of the manual logger,while FIG. 8b shows an example flow chart of the automatic logger.

[0098]FIG. 9 shows an exemplary video frame indexing format that can beused in accordance with the instant invention. FIG. 10 shows therelationship between the video index, feature and event database, inaccordance with one embodiment of the invention.

[0099]FIG. 7a shows an exemplary architecture for the video contentmanagement system of the instant invention. FIG. 7b provides a diagramshowing the signal paths for the embodiment of the video contentmanagement system of FIG. 7a.

[0100]FIG. 8c shows the video indexing in accordance with a preferredembodiment of the instant invention. FIG. 8d shows a search and browsingfunction flow provided by a preferred embodiment of the instantinvention. FIG. 8e shows a flow diagram of the video server. FIG. 8fshows a flow diagram of the video analysis subsystem used in accordancewith the preferred embodiment. It is noted that both FIGS. 8f and 8 cdemonstrate the automatic logging feature of the present invention,wherein audio, video and/or text analysis are used to do automaticlogging. In other words, in accordance with the invention, audio, videoor text analysis (texture, motion, color, shape, keyframe) technologymay be used to perform automatic logging of events. FIG. 8g shows apreferred flow diagram of the application server.

[0101]FIG. 12 shows how the instant invention may be implemented as amultimedia logging and application Web-based user interface. In theWeb-based embodiment, the application is a series of Web pages thatinteract with the user by running different components and modules. Theplatform of the Web application is preferably any Java-enabled browsersuch as Microsoft's Internet Explorer or Netscape Navigator.

[0102] The user interface may be hard coded for each application inwhich the invention is being used. For example, if the system of theinstant invention is designed for a sports application, the userinterface can be hard coded to provide the appropriate buttons and otherinterface objects that correspond to the types of events that one woulddesire to log in connection with the particular sporting event. Ofcourse, each sport would have different needs. Thus, the system could bemanually recoded to provide the appropriate customized interface foreach application.

[0103] On the other hand, in accordance with another aspect of theinvention, Et GUI generator may be provided that will enable theapplication to be customized automatically for any situation or eventwithout the need to make coding modifications. In other words, thesystem is preferably programmed to be customizable on the fly byenabling user interfaces to be automatically generated based on enteredinformation by the user, thereby avoiding the need to hard code theinterfaces. This feature makes the invention very flexible andcustomizable because it is database driven.

[0104] Specifically, the formation of the GUI and the logging databaseis database driven, i.e. is based on information in an administrationdatabase that can be edited using the system. In other words, thedatabase fields in the logging database (as well as the GUI itself) arealso database driven, i.e., they are dynamically generated based on theinformation in the administration database. Thus, there is a dynamicgeneration of the GUI and the logging database structure and propertiesbased on user-defined input. Accordingly, this customization featureprovides a customizable access module (CAM) that builds the GUI and thelogging database. Thus, the GUI generator preferably does all of thefollowing things based on information entered in the administrationdatabase:

[0105] 1) creates the GUI;

[0106] 2) assigns properties to the GUI objects;

[0107] 3) builds a logging database (or expands or modifies an existinglogging database); and

[0108] 4) defines properties for each field in the logging database.

[0109] The following description explains how the instant system is ableto create customized HTML- or XML-based user interfaces for thecollection of standardized information. In accordance with this feature,a user interface generator (UIGenerate) and a GUI applet are providedfor the creation of the customized HTML-based user interfaces.

[0110] Specifically, the GUI applet connects with the UIGeneratorprocedure and passes a series of parameters indicating what tables touse for the creation of the user interface. The GUI applet receives theinformation from the UIGenerater procedure and constructs the graphicaluser interface based on this information. The communication between theGUI Applet and the UIGenerater procedure is preferably based on XML sothat it can be extended as desired. The GUI applet collects the GUIelements selected by the user and transmits them back to the UIGenerator. The UIGenerator connects to both Oracle and MSSQL databaseengines. It retrieves the set of element to be rendered into thegenerated user interface from user-specified tables in the databaseand/or SQL stored procedures. The UIGenerator receives a list ofelements from the GUI applet and inserts that information into anappropriate table in the database. The table where the insertions willtake place is preferably parameter-specified. In addition this table canbe related by a foreign key. Thus, this process can all be done throughthe use of a generic SQL stored procedure. The UIGenerator will connectto a database (Oracle, MSSQL) and retrieve a table from the databasespecified.

[0111] The description of the controls that are stored in the databasetable, should include provision for the behavior of the controls.Namely, the on_click, on_dlb_click function will contain differentbehavior for different controls. For example when one of the controls isclicked, the color of that button should change. This indicates to theuser that the button has been selected. In addition every elementbelongs to a group. This can be a table or frame. If there are nelements that belong to a group g and group g displays only m elementswhere m<n then behavior can be added to the elements of group g. Whenthe user double clicks on one of these elements the element ispreferably replaced by a drop list that gives the choice of changing theelement to any of the elements in group g. If the database tables togenerate the GUI do not use buttons, but some other control, liketextboxes, the behavior should be defined on the component through theuse of the on_click and on_dlb_click functions. The on_click functionchanges the type of font on the selected element, while the on_dbl_clickgives the option of changing the element's value by means of a drop listor enables the user to type in the textbox (enables the textbox forwriting)

[0112] Preferably, the insertion process into the database should nottake more than one second. The GUI applet should return a feedbackstring to indicate the confirmation of the insertion.

[0113] Preferably, there are two tables in the database to define theuser interface. One of the tables defines the Groups (HTML tables, inthis case). The other table contains the elements with their attributes,methods and properties. This component's purpose is to query thedatabase for information regarding the user interface to be generated.

[0114] In this embodiment, the input parameters are:

[0115] 1. Database Name to connect to

[0116] 2. Table or Store procedure

[0117] 3. Parameter Structure

[0118] The table or store procedure will return a recordset with anumber of rows. The component will use the different columns to generatea table of buttons with all the attributes specified in the columns,thereby automatically generating a customized user interface.

[0119]FIGS. 13 and 14 show examples of two tables in accordance withthis GUI generator embodiment. The resulting output of these tables bythe UI generator is shown in FIG. 15. It is noted that the servletshould be extensible so that one can add more parameters to the databaseas desired. For example, if one wanted to add the color attribute to theHTML object (Button in this case) one would add a column named color. Inaddition, functionality such as client script and behavior can be added.In one embodiment, the whole description of the object could be just astring. For example: After the Cust Name field, one could have a stringfield named HTMLstring:

[0120] <INPUT type=“submit” value=“Ok” id=OK name=“Ok”LANGUAGE=javascript onclick=“return submit1_onclick( )”>

[0121]FIG. 16 shows a high level architecture of the components of thisaspect of the system. FIG. 17 shows an overall flow diagram of this GUIgenerator feature.

[0122] While the use of a GUI applet has been described above, this userinterface generator feature may alternatively be implemented with theuse of a standard Web browser. In this alternative embodiment, thebrowser communicates with the servlet, i.e. a session is establishedbetween the browser and the servlet.

Example Basketball Implementation

[0123] FIGS. 18-47 show exemplary screen shots for a basketballimplementation of the instant invention. A more complete understandingof the operation of the instant invention is provided by this example.

[0124] As shown in FIG. 18, to launch the application it is necessary tofirst “log in”. In other words, the application includes certainsecurity features to assure that only authorized persons have access tothe system. Thus, the person logging (“the logger”) must have accessprivileges to navigate through the system. To safe guard content, asystems administrator preferably has the sole discretion to permit ordisallow an individual access to “their” content or media. The customerdetermines who shall be the administrator (e.g. a producer, director,associate director, intern, etc.) The administrator will usually grantaccess to a producer, an assistant director or intern.

[0125] As shown in FIG. 19, when the system administrator has grantedaccess, the “Game Wizard” will appear. At this page the user is able toselect a game that has previously logged events (to continue logging ataped game). Or select “New” to log a “live” event (as it occurs in realtime).

[0126] As shown in FIG. 20, if the user chooses to log a “New” or “Live”game, he must enter the information requested in the window. Thisinformation is available and based on the team's itinerary. The data foreach field is required to assist in the search and retrieval functionsof the logging application.

[0127] As shown in FIG. 21, if the user desires to resume a previouslylogged game or event, then he must “Select a game or event date”. Asshown in FIG. 22, upon “Selecting a game date”, the logger can use thepull down menu's to access the “Home and Away Teams”, and then click on“next” to proceed.

[0128] As shown in FIG. 23, after clicking on “Next” on the previousscreen, a “Players Roster” will appear and indicate that this “Game HasLogged Events”. The user then Selects the players in the active roster.Enter a “New Player”, if the player was acquired by draft, trade or anyother means by which a player joins a team.

[0129] As shown in FIG. 24, if a player is acquired by draft, trade, orother means, the administrator or logger must enter the player'sprofile: Team, first & last name, player number, position (guard,forward, or center), height, weight, and even web address or e-mail. Itis on this page that the logger can delete a preseason player (a playeron a 10-day contract), or update a player's weight, etc. A player orperson cannot actually be “deleted” but the person or player will bedesignated as “inactive”, if delete is desired. After the persons orplayers have been traded, drafted, updated or designated “inactive”, theuser can save and close the players administration window. Then, byclicking on “confirm” the logging page will appear.

[0130]FIG. 25 shows a user interface for the logging of events. This isa main and important screen in accordance with the instant invention, inthat it is used for all logging of events by the logger. As explainedabove, this screen can either be hard coded or automaticallygenerated/customized by the GUI generator. The logging page is where thelogger records, indexes, and catalogues any media event. The applicationincludes check-boxes, radio buttons, push buttons, text boxes, circlesand/or a touch screen. In accordance with a preferred embodiment of theinvention, all of the interface objects on this interface arecustomizable for the particular application being logged.

[0131] As shown in FIG. 26, if applicable, to begin logging an event,the user selects a person(s), nature of the event (2 point shot), inputor “feed” sources (cameras: cat, Pink, Green), and designates whetherthe particular event was a “highlight”, “Blooper”, or provides his owncategory or description. In this basketball example, virtually everypossible scenario is covered on the logging application. “Get Time” isdesigned to jot down or grab the anticipated time of an event orhappening while it is developing. If the event developed like the loggeranticipated, then he would select “Add Event” to store the selectedevent in the database. If the logger made an error after “Adding anEvent” (e.g. he recorded a 3 point shot and the referee designated it a“2 point shot”) he can simply click on “Delete Event” and this willremove only the previously entered event. Getting a report of the loggedevent is very simple—just click on “Quick Report”.

[0132] As shown in FIG. 27, after making your selection, the applicationwill ask the logger to confirm the request. This is a safety feature toensure the accurate inputting of data or events.

[0133] As shown in FIG. 28, a “Quick Report” can be selected to give thelogger and the production team instant access to shared informationacross a LAN (local area network) or WAN (wide area network) across theinternet.

[0134] In the content management arena, “content is king” but any typeof clock (timecode, military clock, stop-watch, any time-zone or anyestablishing point of reference) is the “life-blood” of the content. Theclock can be “real time or a user defined time”. Whatever form it takesin order to retrieve this information universally, this has to be thebasis for which events are indexed and retrieved. Setting the clock isas simple as using the application, just point and click. A logger canuse a clock format, military time format or whatever format applicableto the user, as seen in FIG. 29.

[0135] As shown in FIG. 30, at the completion of the event, the loggercan point and click on reports to review the event. Reports can beretrieved in several ways: by Date, Game, Player or Team. The reportsare intelligent, in that users can pinpoint, search and retrievespecific data (text, audio & video) via specific dates, occurrences(regular or irregular), irrespective of a player's current location,team, status, or history. The data is eternally associated with a person(player), place (city/state/country) or thing (event, occurrence oract).

[0136]FIG. 31 shows a report that is the result of a “Player” search. Ifa person has been entered into the database, the person and hisassociated events (whether active or inactive) are searchable,retrievable and stored for recollection at a later date.

[0137]FIG. 32 shows a report that is the result of a “Player”search—Player: Allan Houston of the NY Knicks. It is an abbreviatedversion of Allan Houston's events over the course of 4 games. Theability to search is possible by various parameters: Game number, Date,Team events, Game Events or Game comments.

[0138]FIG. 33 shows a report that is the result of a “Team” search: theNY Knicks. Searches can be accomplished by game date or by game. Gamedate will search all the games or events that occurred on thatparticular date.

[0139] The report of FIG. 34 is the result of a “Games” search. Thisreport shows all the games in the database that are available for“Online” on-demand retrieval. The games lists search displays all thegames on the database, either in ascending or descending order. Gamenumbers, game dates, teams, events and comments are all searchable andretrievable. The menu will display all customized fields according to auser's discretion, e.g. medicine, legal, technological or sports. Thereports are not limited to sports, but are as diverse as the mediamanagement application and it's intelligent indexing and search agentsprovided by the application.

[0140]FIG. 35 shows a report that is the result of an “Individual Game”search: between the Boston Celtic and the New York Knicks. The reportdisplays each event logged: the player involved in the event, the typeof event or function performed by the player, relative score of the“Home team” for that particular event, the associated timecode, clock,user clock, stop watch or time-zone (as the event occurred), “Visitingteam points”, visiting team events, visiting team players, a “commentssection” for unusual or incidental occurrences that require a detailedexplanation of the occurrence, performance or event, and camera anglesor input source routing information.

[0141]FIG. 36 shows the system administration menu and consists of itemsthat define the database and logging features. Such features include:Games Admin., Teams Admin., Players Admin., Locations Admin., EventsAdmin., and Forced Suggestion.

[0142] As shown in FIG. 37, Games Administration, is where anadministrator or logger enters the itinerary of the desired team(s).This information is linked to past and future information, for instantupdates and historical data, by which users can access (via a link ordownload) Game information, such as previous games and long standingrivalry's.

[0143] As shown in FIG. 38, Teams Administration, is where anadministrator or logger enters the team profile. E.g. Team name: NewYork Knicks/League: NBA/Contact Person: David Stem/Address: 2Pennsylvania Plaza, NY, N.Y. 10010/URL: www.nyknicks.com. It is notedthat the “real estate” on any screen is available for: Title sponsors,Advertisements, segmented information for the purpose of Tourism, Sportsbars, Restaurants, and Hotels.

[0144] As shown in FIG. 39, Players Administration, is where anadministrator or logger enters the players profile. Eg. Team name: LosAngeles Lakers/First name: Shaquille/Last name: O'Neal/Player number:34/Position: Center/Height: 7-3/Weight: 305/URL: www.Dunk.net. The LRL(Uniform Resource Locator) provide links to player's web sites and canupdate and retrieve personal web site links, associated press int'lheadlines, up to the minutes stats and virtually any internet accessibleinformation globally available.

[0145] As shown in FIG. 40, Location Administration is where anadministrator or logger enters the cities/locations that host a NBAteam.

[0146] As shown in FIG. 41, Events Administration permits theadministrator or logger to customize the logging application with theirown special terminology. For example: the button “2 pt. Shot” can bechanged to “jumper”. Or “dunk” can be customized to say “Jam!”Preferably, every item on the application is totally customizable,allowing the user total freedom to record a sporting event, fashionshow, courtroom proceeding, or any desired A/V event.

[0147] As shown in FIGS. 41 and 42, Events Administration permits theadministrator or logger to customize the logging application with theirown special terminology. For example: the button “2 pt. Shot” can bechanged to “jumper”. Or “dunk” can be customized to say “Jam!” Everyitem on the application is totally customizable, allowing the user totalfreedom to record a sporting event, fashion show, courtroom proceeding,or any A/V event.

[0148]FIG. 43 demonstrates a “Forced Suggestion” feature of the instantinvention. This is the selection process for designating which buttonsor boxes will have the “Forced Suggestion” feature. “Forced Suggestion”is a feature that prompts the end user to go beyond the basicdescription of an event. A user can record an event such as a fashionshow or court proceeding and while indexing the typical events, the usercan program certain sequences of clicks to ask further question to thelogger, e.g. “Is the subject smiling? Sad? Furious? Or what is theclimate of the courtroom?” This way of logging compels the user to bemore specific in the task at hand.

[0149] As seen in FIG. 44, Forced Suggestion is a feature that “reminds”the logger to complete additional steps based on the predeterminedbuttons that the administrator designates. For example: If a playerdunks a shot over an opponent, did the player react after the dunk? Didhe raise his fists in victory? Did he celebrate? These are forcedsuggestions, which makes the logger submit more details on a particularevent.

[0150]FIG. 45 shows an exemplary “Clips” feature of the invention inconnection with a logged hockey game. Any of the listed events in FIG.45 can be clicked-on to see the actual video segment of the event. Inaddition, the system enables the concatenation of multiple selectedmedia segments into a single media (e.g., video) stream, if desired.Thus, in accordance with the invention, a user can easily search for,retrieve and view captured video segments after they have been logged.FIG. 46 illustrates a video segment being shown in response to clickingon an event listed on FIG. 45.

[0151] As can be seen from the above description, the invention providesa media content management system that significantly improves loggingoperations, as well as provides a sophisticated search and retrievalsystem for media assets. The system can be used in a variety ofapplications and by various people for various reasons within eachapplication. For example, during a sporting event, numerous loggerscould be using the system in a networked embodiment. Each of the loggerscould have a customized interface to suit their specific logging task.All of the logged events, and optionally the associated media segments,can then be stored to a database for later search, retrieval and viewingby anyone with authorized access to the system. In this manner, mediaassets can be easily and efficiently managed.

[0152] While the invention has been described in connection with what ispresently considered to be the most practical and preferred embodiments,it is to be understood that the invention is not to be limited to thedisclosed embodiments, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

What is claimed is:
 1. A media logging system for indexing media,comprising: a timer object that provides a time reference upon requestin connection with the media; and a logger object that logs predefinedevents that occur in the media by associating the events with respectivetime reference from the timer object.
 2. The logging system of claim 1,wherein the logger object includes a graphical user interface having aplurality of interface objects that can be selected by a user of thelogging system in order to log events that occur in the media.
 3. Thelogging system of claim 2, wherein the interface objects include labelsthereon corresponding to predetermined events that may typically occurin the particular media being logged.
 4. The logging system of claim 2,wherein the logging system is adapted for use in logging a sportingevent and the interface objects correspond to events that occur duringthe sporting event
 5. The logging system of claim 1, wherein thegraphical user interface is customizable to correspond to types ofevents that occur in the particular media being logged.
 6. The loggingsystem of claim 1, wherein the system includes an arrangement thatautomatically logs predefined events in the media being logged based onvideo analysis.
 7. The logging system of claim 1, wherein the systemincludes an arrangement that automatically logs predefined events basedon text analysis.
 8. The logging system of claim 1, wherein the systemincludes an arrangement that automatically logs predefined events basedon audio analysis.
 9. The logging system of claim 1, wherein the systemincludes a video server that captures logged events and digitally storedthe captured events as media segments.
 10. The logging system of claim9, wherein the user interface includes a feature that enables the mediasegments to be selectively retrieved and viewed.
 11. The logging systemof claim 10, wherein the feature includes a search engine that enableslogged events to be searched using various search parameters.
 12. Amedia asset management system for managing media assets, comprising: alogging application that creates indexes of events that occur in themedia; a video server that captures and digitally stores events loggedby the logging application as media segments; and a search and retrievalengine that enables the media segments to be located and retrieved basedon the indexes.
 13. The media asset management system of claim 12,wherein the logging application includes a graphical user interface thatenables a user to select predefined events that occur in the media to beindexed.
 14. The media asset management system of claim 13, wherein thegraphical user interface includes interface objects having labelsthereon corresponding to predefined events that typically occur in theparticular media being logged by the logging application.
 15. The mediaasset management system of claim 14, wherein the logging application isadapted for use in logging a sporting event and the interface objectscorrespond to events that occur during the sporting event.
 16. The mediaasset management system of claim 12, wherein the video server includesan arrangement that automatically logs predefined events based on videoanalysis.
 17. The media asset management system of claim 12, herein thevideo server includes an arrangement that automatically logs predefinedevents based on text analysis.
 18. The media asset management system ofclaim 12, wherein the video server includes an arrangement thatautomatically logs predefined events based on audio analysis;
 19. Themedia asset management system of claim 13, wherein the user interfaceincludes a feature that enables the media segments to be viewed.
 20. Acustomizable media asset management system for managing media assets,comprising: a logging application that creates indexes of events thatoccur in the media; a video server that captures and digitally storesevents logged by the logging application as media segments; and agraphical user interface generator that creates customized userinterfaces for the logging application based on information entered in adatabase.
 21. The customizable media asset management system of claim20, wherein the graphical user interface generator generates userinterface objects for a logging screen for use by a user when using thelogging application.
 22. The customizable media asset management systemof claim 20, wherein the creation of the customized user interface isdatabase driven.
 23. The customizable media asset management system ofclaim 20, wherein the user interface generator is further operable todynamically generate a logging database for use by said loggingapplication.